Skin and Motion Cues Incorporated Covariance Matrix for Fast Hand Tracking System

Hand tracking is one of the essential elements in vision based hand gesture recognition system. The tracked hand image can provide a meaningful gesture for more natural ways of Human Computer Interaction (HCI) systems. In this paper, we present a fast hand tracking method based on the fusion of skin and motion features incorporated covariance matrix. First, hand region is detected using a fusion of skin and motion cues, and a region of interest (ROI) is created around the detected region. During the tracking, skin and motion features are extracted around top, left and right corners of the ROI and hand displacement is measured using ROI based tracker. To increase the robustness, we incorporate a covariance matrix of the ROI window as a region descriptor to represent the target object. In the consecutive frames, we measure the distance descriptor covariance matrix (DDCM) between the target object and the covariance matrix extracted from new ROI position. When DDCM is not satisfying a certain acceptable threshold, ROI position is adjusted by shifting the ROI window around nearest neighbor to obtain a set of candidate regions. We assign a candidate region which has the smallest DDCM as the correct estimated ROI position. The experimental result shows that our approach can track the hand gesture under several real-live scenarios with a detection rate above 95% with the tracking speed at 42fps on average.